Systems and methods for electronic mapping and localization within a facility

ABSTRACT

Systems and methods for electronically mapping a facility are presented. The method comprises obtaining a CAD file that includes graphical representations of a facility. An occupancy-map image is generated based on the CAD file. A sensor, such as a sensor on a self-driving vehicle, is used to detect a sensed feature within the facility. Based on the sensed feature, the occupancy-map image can be updated, since the sensed feature was not one of the known features in the CAD file prior to the sensed feature being detected by the sensor.

REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. Provisional PatentApplication No. 62/515,744, filed 6 Jun. 2017, the contents of which areincorporated herein by reference.

TECHNICAL FIELD

The described embodiments relate to systems and methods for mapping afacility, and in particular to electronically mapping and remapping afacility for the autonomous navigation of a self-driving vehicle.

BACKGROUND OF THE INVENTION

Simultaneous localization and mapping (“SLAM”) is a technique thataddresses the computational problem of simultaneously mapping anenvironment while localizing with respect to the map. For example, SLAMcan be used in robotics so that a robot can generate a map of a facilityfor the purposes of navigation while continuously orienting itself withrespect to the map. Applications for SLAM are widespread. SLAM can beused in facilities such as hospitals, warehouses, manufacturing andindustrial facilities, retail locations, etc.

Known SLAM techniques require that an entire facility be traversed (e.g.by a robot) in order to generate a map of the facility. For largefacilities, traversing the entire facility in order to generate a mapcan be an onerous task. In some cases, it may be necessary to obtain amap of the entire facility, or substantial portions thereof, before arobot can be used to perform other functions in the facility.

Some solutions have been proposed for providing a map without therequirement for traversing an entire facility or environment. Forexample, an electronic map for use in automobile navigation may begenerated based on images of the Earth captured by a satellite. Inanother example, an electronic map for use by vehicles in an indoorindustrial facility may be generated based on a schematic of thefacility. However, in these cases, a new set of satellite images or anew schematic of the facility are required in order to update the map toreflect any changes in the environment or facility.

SUMMARY OF THE INVENTION

In a first aspect, there is a method for electronically mapping afacility. The method comprises obtaining a CAD file that has graphicalrepresentation representing known features of the facility. The CAD filemay be obtained from a computer system. An occupancy-map image isgenerated based on the CAD file. A sensor is used to detect a sensedfeature in the facility, and the occupancy-map image is updated based onthe sensed feature. The sensed feature is a feature that is not part ofthe known features in the CAD file prior to the sensed feature beingdetected.

According to some embodiments, the method further comprises detecting aknown feature with the sensor, and locating the sensor relative to theoccupancy-map image based on detecting the known feature.

According to some embodiments, generating the occupancy-map imagecomprises a rasterization based on the CAD file.

According to some embodiments, the CAD file comprises graphical objects,and the rasterization is based on the graphical objects.

According to some embodiments, the CAD file further comprisesnon-graphical objects, and the method further comprises storing a mapfile on a computer system, comprising the occupancy-map image, thegraphical objects, and the non-graphical objects.

According to some embodiments, the method further comprises generating akeyframe graph based on the CAD file, and storing a map file on themedium. The map file comprises the occupancy-map image and the keyframegraph.

According to some embodiments, the sensor is in communication with acontrol system of a self-driving vehicle, and the method furthercomprises storing the occupancy-map image on the control system of theself-driving vehicle and navigating the self-driving vehicle based onthe occupancy-map image.

In a second aspect, there is a method for locating a sensor in afacility. The method comprises obtaining a CAD file from anon-transitory computer-readable medium that includes graphicalrepresentations representing known features of the facility. A keyframegraph is generated based on the CAD file, which includes a plurality ofkeyframes, each associated with a pose. A feature within the facility isdetected using the sensor, and the sensed feature is then compared withthe plurality of keyframes. Based on matching the sensed feature with amatched keyframe, a current pose of the sensor is determined. Thematched keyframe is one of the plurality of keyframes, and the currentpose is the pose associated with the matched keyframe.

According to some embodiments, the method further comprises detecting anew feature in the facility with the sensor, and updating the keyframegraph based on the new feature. The known features of the facility donot comprise the new feature prior to detecting the new feature.

According to some embodiments, generating the keyframe graph comprisesdetermining the plurality of poses based on the range of the sensor.

In a third aspect, there is a system for electronically mapping afacility. The system comprises a sensor for detecting a sensed featureof the facility, a non-transitory computer-readable medium storing a CADfile, and a processor in communication with the sensor and the medium.The CAD file includes graphical representations representing knownfeatures of the facility. The processor is configured to generate anoccupancy-map image based on the CAD file, store the occupancy-map imageon the medium, and update the occupancy-map image based on the sensedfeature. The sensed feature is a feature that is not part of the knownfeatures in the CAD file prior to the sensed feature being detected.

According to some embodiments, the sensor is on a self-driving vehicle,and the medium and processor are part of a control system on theself-driving vehicle. The control system of the self-driving vehiclestores the CAD file and generates the occupancy-map image.

According to some embodiments, there is a second non-transitorycomputer-readable medium and a second processor that are part of aserver. The server is in communication with the self-driving vehicle.The CAD file is stored on the second medium, and the second processor isconfigured to generate the occupancy-map image. The server transmits theoccupancy-map image to the self-driving vehicle.

According to some embodiments, the CAD file is stored on anon-transitory computer-readable medium that is not the second medium(i.e. is not stored on the server), and the occupancy-map image is notgenerated by the second processor (i.e. is not generated by the server).

According to some embodiments, the sensor is a LiDAR device.

According to some embodiments, the sensor comprises a vision system.

In a third aspect, there is a method for displaying an electronic map.The method comprises receiving a map file from a non-transitorycomputer-readable medium. The map file comprises graphical objects, anoccupancy-map image associated with the graphical objects, andnon-graphical objects associated with the graphical objects. The methodfurther comprises displaying a rendering of the map file on a computerdisplay, wherein the rendering comprises a graphical rendering of theoccupancy-map image and textual rendering of the non-graphical objects.

According to some embodiments, the graphical rendering of theoccupancy-map image comprises spatial features, and the textualrendering of the non-graphical objects comprises spatially locating textin relation to the spatial features based on the association between thespatial features, the graphical objects, and the non-graphical objects.

BRIEF DESCRIPTION OF THE DRAWINGS

A preferred embodiment of the present invention will now be described indetail with reference to the drawings, in which:

FIG. 1 is a system diagram of a system for electronically mapping afacility according to some embodiments;

FIG. 2 is a visual representation of a CAD file of a facility, such asmay be used by the system shown in FIG. 1;

FIG. 3 is a visual representation of an occupancy-map image, such as maybe originally generated based on the CAD file shown in FIG. 2;

FIG. 4 is a visual representation of a key-frame graph such as may beoriginally generated based on the CAD file shown in FIG. 2;

FIG. 5 is a visual representation of the facility in FIG. 1 asdetermined by the sensors of a self-driving vehicle;

FIG. 6 is a visual representation of a map file comprising the CAD fileof FIG. 1 and an associated occupancy-map image; and

FIG. 7 is a flow diagram depicting a method for electronically mapping afacility, according to some embodiments.

DESCRIPTION

Referring to FIG. 1, there is a system 100 for electronically mapping afacility, according to some embodiments. The system comprises a sensor110 in communication with a local computer system 112, and remotecomputer systems including a server 124 and a workstation 126. Thecomputers 112, 124, and 126 are in communication using variousnetworking equipment and communications links 128, which may includeknown protocols such as Ethernet and WiFi (e.g. IEEE 802.11), andcellular-based schemed such as 3G, LTE, which can be used overlocal-area networks (LANs), wide-area networks (WANs), the Internet,intranets, and mobile-device networks. As such, the computer 112 may beconsidered “local” with respect to the sensor 110, while the server 124and the workstation 126 may be remotely located with respect to thesensor 110.

According to some embodiments, the sensor 110 can be part of (or mountedon) a self-driving vehicle 130, and the local computer system 112 can bepart of the vehicle's control system. According to some embodiments, theserver 124 may comprise a fleet-management system for administering andproviding other functions related to a self-driving vehicle or a fleetof self-driving vehicles.

The computer 112 (e.g. as may be part of a self-driving vehicle controlsystem) comprises a processor 114, memory 116, and a non-transitorycomputer-readable medium 118. The non-transitory computer-readablemedium 118 may be used to store computer-readable instruction code thatcan be executed by the processor 114 using the memory 116 in order toconfigure the processor 114 to perform the steps of the various methodsand processes described herein. The server 124 and the workstation 126similarly each comprise their own processors, memory, and non-transitorycomputer-readable media (not shown). According to some embodiments, anyor all of the methods and processes described herein (or parts thereof)may be performed by any of the computer 112, the server 124, and theworkstation 126.

According to some embodiments, the sensor 110 may be a LiDAR device (orother optical/laser, sonar, or radar range-finding sensor).

According to some embodiments, a self-driving vehicle 130 may includemore than one sensor and more than one type of sensor. In the exampleshown, the self-driving vehicle 130 includes a front sensor 110 fordetecting objects in front of the vehicle, and a rear sensor 122 fordetecting the objects behind the vehicle.

Additionally, or alternatively, the vehicle 130 may also include sensors120 a and 120 b, which may be optical sensors, such as video cameras.According to some embodiments, the sensors 120 a and 120 b may beoptical sensors arranged as a pair in order to provide three-dimensional(e.g. binocular or RGB-D) imaging. Similar sensors may also be includedon the rear, sides, to, or bottom of the vehicle (not shown). All of thesensors 110, 120 a, 120 b, and 122 may be generally or collectivelyreferred to as “sensors”.

In the example of a self-driving vehicle, the computer 112 may be partof the vehicle's control system that is in communication with thesensors 110, 120 a, 120 b, and 122. The control system may store anelectronic map of the vehicle's environment in the medium 118. Thecontrol system may use the electronic map, along with input receivedfrom the sensors in order to autonomously navigate the vehicle. Forexample, the control system, along with the sensors, may implement asimultaneous localization and mapping (“SLAM”) technique in order to mapthe vehicle's environment and to locate the vehicle (i.e. locate thesensors) relative to the stored electronic map.

Referring to FIG. 2, there is shown a visual representation of acomputer-aided design (“CAD”) file of a facility 200, according to someembodiments. CAD files may be generated using computer software thatallows a human operator to create two-dimensional and three-dimensionaldrawings and models. For example, a CAD file can be used to draw thefloorplan of an industrial facility. Generally, CAD files may be storedin a human-readable format such as ASCII text. CAD files may comprisegraphical objects that represent spatial features that can be renderedand displayed by a computer.

Graphical objects may define where the object should be displayed (i.e.in space), the color of the object, the dimensions of the object, theshape of the object, etc. Non-graphical objects may provide additionalinformation associated with the CAD file as a whole, or with specificgraphical objects. For example, a name associated with a graphicalobject may be stored as a non-graphical object.

As shown in FIG. 2, the CAD file may be displayed in respect ofgraduations (shown as a dot grid) in order to assist a human who iscreating, editing, or viewing the CAD file to understand the scale anddimensions of the objects in the CAD file.

The CAD file 200 includes graphical objects 210, 212, 214, 216, and 218,which relate to the features of the facility. In other words, the CADfile 200 represents a layout of the floor of the facility, which showswhere particular features are located.

For example, the feature represented by the graphical object 210 may bea shelf, such as an inventory shelf. In association with the graphicalobject 210 is the textual rendering of a non-graphical object 232. Asshown, the non-graphical object 232 includes the information “Shelf”,and the non-graphical object has been rendered in order to display theword “Shelf” over the graphical object 210.

Similarly, a non-graphical object 234 includes the information “AssemblyArea”. In the example shown, the text “Assembly Area” is not necessarilyrendered (shown) with respect to the visual representation of the CADfile 200. Rather, the information pertaining to the non-graphical object234 is shown in FIG. 2 to indicate that “Assembly Area” may beinformation (e.g. metadata) associated with the graphical object 216.

Referring to FIG. 3, there is shown a visual representation of anoccupancy-map image 300 for the sake of description and explanation. Theoccupancy-map image 300 corresponds to the upper-left corner of the CADfile 200 shown in FIG. 2, including correspondence to the feature 210.

The term “image” is used herein with respect to “occupancy-map image”since an occupancy-map image is a computer file that can be representedin a raster format. According to some embodiments, the occupancy-mapimage may be in the form of a bitmap file. According to someembodiments, the occupancy-map image may be in the form of a portablegreymap (“PGM”) file.

Generally, the occupancy-map image file may be defined in terms ofindividual pixels, points, or regions determined by a discrete location,and with a corresponding value assigned to each discrete location. Asshown in FIG. 3, the occupancy-map image 300 comprises a grid thatdefines discrete squares (pixels), such as the pixel 302 in the topright corner.

For example, and in respect of FIG. 3, locations can be defined in termsof x-y coordinates. As such, the bottom row of pixels in theoccupancy-map image 300 may have the (x,y) coordinates (0,0), (1,0),(2,0), etc., from left and counting to the right. Similarly, theleft-most column may have the (x,y) coordinates (0,0), (0,1), (0,2),etc., from the bottom and counting upward.

A value is assigned for each pixel in the occupancy-map image. Accordingto some embodiments, the values may correspond to states of “occupied”,“not occupied”, and “unknown”. In the example shown FIG. 3, each ofthese states is represented by the value 1, 0, and −1 respectively. Forthe sake of explanation and to enhance visual perception, the pixelcells in FIG. 3 are shown with a specific border based on their value.

For example, and in regards to the coordinate system described above,the occupancy-map image 300 includes pixels with (x,y,value) of(0,6,−1), (1,6,0), (2,6,0), (3,6,1), and (4,6,−1). Referring back toFIG. 2, the pixels (0,6,−1) represents the left-most part of thefacility, which has not yet been mapped, and is thus “unknown”. Thepixels (1,6,0) and (2,6,0) represent the “unoccupied” (and known) areato the left of the feature 210. The pixel (3,6,1) represents theboundary of the feature 210 and is thus “occupied”. The pixel (4,6,−1)represents an “unknown” area that is internal to (i.e. enclosed by) thefeature 210.

According to some embodiments, pixels that are internal to (i.e.enclosed by) a feature may be assigned a value of “unknown”, since asensor may not be able to determine what is beyond (e.g. behind) theboundary of the feature. According to some embodiments, when theoccupancy-map image file is being generated or updated, it may bepossible to determine that a feature encloses a region, and thereforeset all of the internal pixel values as “occupied”. According to someembodiments, pixels that are internal to (i.e. enclosed by) a feature byassigned a value of “occupied” based on the definition of the graphicalobjects in the CAD fie even though the sensors may not be able to detectobjects internal to the physical feature.

Generally, an occupancy-map image may be used with known SLAMtechniques. For example, the occupancy-map image may be used withrespect to the mapping and re-mapping aspects of known SLAM techniques.

Referring to FIG. 4, there is shown a visual representation of akeyframe graph 400 according to some embodiments. Similar numerals areused in FIG. 4 as in FIG. 2 in order to label similar or correspondingelements.

Similar to the occupancy-map image previously discussed, the keyframegraph may be used with known SLAM techniques. For example, the key-framegraph may be used with respect to the localization aspects of known SLAMtechniques. Generally, keyframes can be used as a reference point thatcan serve as the basis for localization using visual SLAM techniquesand/or integrated inertial-visual SLAM techniques.

The keyframe graph 400 includes multiple poses, represented by dots inFIG. 4, such as the pose 436, at which a keyframe is determined.According to known SLAM techniques, keyframes may be determined at anyarbitrary pose, by gathering sensor data at the pose. The population ofkeyframe poses in a graph may be relatively sparse or dense according tothe particular technique used. For example, keyframes may be generatedby moving a sensor throughout a facility, and gathering sensor data atposes defined in terms of time (e.g. periodically), displacement, orother events.

According to some embodiments, SLAM techniques can be employed withoutmoving the sensor from one pose to another, by generating a keyframegraph from a CAD file. According to some embodiments, the keyframe graphay be generated based on the graphical objects in a CAD file. In otherwords, a keyframe graph can be generated from a CAD file so that SLAMlocalization techniques can be employed, without having to firstgathering sensor data from a series of poses.

FIG. 4 uses an example of a sensor on a self-driving vehicle 420 for thesake of description. In the example, the sensor on the self-drivingvehicle 420 has a maximum range as indicated by the dashed line 438.According to some embodiments, simulated keyframe poses may bedetermined based on a location transform of the graphical objects in theCAD file based on the range of the sensor. In other words, if the rangeof the sensor is known or can be assumed, then the simulated poses usedfor generating the keyframes can be located at a distance from aparticular feature (e.g. the feature 416) based on the sensor range, inorder to produce a relatively sparse keyframe graph.

According to some embodiments, it is not necessary to generate akeyframe for a pose that is located at a distance greater than the rangeof the sensor from any feature defined by a graphical object in the CADfile.

According to some embodiments, when there are two features separated bya distance that is less than the range of the sensor, then a keyframemay be generated at a pose that is half-way between each feature. Forexample, the pose 440 is located equidistant from the features 410 and412.

Once a simulated pose has been determined for a keyframe, then theassociated keyframe data (i.e. simulating the data that would otherwisebe gather by the sensor at the simulated pose) can be determined basedon the graphical objects in the CAD file, thereby generating an initialkeyframe graph that can be used for localization with SLAM.

Subsequently, the keyframe graph can be updated as a sensor is movedthrough the facility, according to known SLAM techniques.

Referring to FIG. 5, there is shown a visual representation of afacility 500 as determined by a self-driving vehicle 520. Similarnumerals are used in FIG. 5 as in FIG. 2 in order to label similar orcorresponding elements.

The self-driving vehicle 520 includes a front sensor 522 and a rearsensor 524. According to some embodiments, the self-driving vehicle 520may have any number of sensors, such as a front sensor only, multiplefront sensors, a rear sensor only, sensors for scanning in threedimensions, side sensors, etc.

The features 510, 512, 514, 516, and 518 represent objects that havebeen detected by at least one of the sensors 522 and 524 as the vehicle520 moves through the facility 500. As the vehicle 520 moves through thefacility 500, it is periodically, arbitrarily, or continuously deemed tobe in a “pose”. A pose is defined as a particular location andorientation in the facility 500 (e.g. a location defined in terms of an(x,y) coordinate and an angle of rotation or direction, e.g. yaw).

At each pose, the sensors 522 and 524 perform a scan in order to detectobjects within a field-of-view. The scan can be understood in terms of aset of lines projecting from the sensors at any particular time acrossthe sensor's viewing angle. The scan lines can be understood ascorresponding to discrete pixels (e.g. the pixels in the occupancy-mapimage as previously described with respect to FIG. 3). For a particularscan line, a sensor (e.g. the sensor 522) measures the distance to adetected object. The distance can be determined in terms of pixels. Whenan object is detected, the pixel corresponding to the object's distanceis deemed to be “occupied”. The pixels between the sensor 522 and theobject are deemed to be “unoccupied”. The pixels on the other side ofthe object from the sensor 522 are deemed to be “unknown” with respectto the particular scan.

At each pose, multiple scans may be performed across a viewing angle,and the scans from multiple poses are accumulated in order to produce animage representing the plane of the facility 500. (Each scan on its ownmay only represent a line). Since there may be some variance and/ortolerance and/or noise or other errors involved for a particular scan,when multiple scans from multiple poses are accumulated, the boundariesor edges for a particular object may vary from one pose to another. Assuch, the boundaries of the features 510, 512, 514, 516, and 518 areshown to be somewhat “blurry”, and not necessarily defined by a single,straight line (e.g. as compared to the CAD file in FIG. 2).

The dashed lines 526 indicates the objects detected by the front sensor522, and the dashed line 528 indicates the objects detected by the rearsensor 524 for the scans that are performed at the particular pose shownfor the vehicle 520 in FIG. 5.

In the example shown in FIG. 5, it is assumed that the vehicle 520 hasalready traversed the entire facility 500 at least once, and, thus, thefacility 500 is general known to the vehicle 520. Conversely, if thepose shown in FIG. 5 was the first pose (i.e. location of first scans)for the vehicle 520, then the only information known to the vehicle 520would be the dashed lines 526 and 528 and the area between the vehicle520 and the dashed lines 526 and 528.

In other words, the visual representation of the facility 500 asdetermined by the self-driving vehicle 520 provides the information usedin generating or updating an occupancy-map image, such as theoccupancy-map image 300 shown in FIG. 3. According to some embodiments,an occupancy-map image may be generated based on a CAD file, and thensubsequently updated based on scans by the sensors 522 and 524, therebyenabling the ability for the sensors to “remap” the facility on anongoing basis as the vehicle moves through the facility.

As shown in FIG. 5, the front sensor 522 of the vehicle 520 has detectedan object 530. As compared to the CAD file shown in FIG. 2, it isimportant to note that the object 530 does not correspond to a featurein the CAD file. In other words, the object 530 may represent a physicalobject that was omitted from the CAD file, or that has been placed inthe facility 500 or moved at a point in time after the creation of theCAD file.

Referring to FIG. 6, there is shown a rendering of an electronic map600, for example, as may be displayed on a computer display. Theelectronic map 600 comprises both the CAD file from FIG. 2 and theoccupancy-map image formed in relation to the visual representation inFIG. 5. Similar numerals are used in FIG. 6 as in FIG. 2 and FIG. 5 inorder to label similar or corresponding elements.

According to some embodiments, the map file (on which the renderingshown in FIG. 6 is based) may comprise a CAD file (or any componentsthereof) and an occupancy-map image file (or any components thereof).According to some embodiments, the CAD file may comprise graphicalobjects and non-graphical objects.

As shown in FIG. 6, the features 610, 612, 614, 616, and 618 may berendered based on the CAD file and/or the occupancy-map image. Forexample, the graphical objects of the CAD file may be the basis for thegraphical rendering of the features 610, 612, 614, 616, and 618. In theevent that a vehicle (or sensor) has not traversed the facility in orderto map the facility, then the CAD file may serve as the only basis forthe graphical rendering of the features 610, 612, 614, 616, and 618.

According to some embodiments, the graphical rendering of the featuresmay be based on the occupancy-map image. The occupancy-map image may begenerated based on the CAD file, and, subsequently, updated based onvehicle sensor scans.

Following on the examples described in respect of FIG. 2 and FIG. 5, ifit is assumed that the feature 630 was not represented in the CAD file,then the graphical rendering of the feature 630 is based on vehiclesensor scans.

The features shown in FIG. 6 are indicative of graphical renderingsbased on the graphical objects in a CAD file as well as vehicle sensorscans. Generally, graphical renderings based on graphical objects in aCAD file are more likely to present straight lines and “crisp” edges,whereas graphical renderings based on vehicle sensor scans may present“blurry” lines and less-definite edges.

According to some embodiments, textual (or other) renderings based onnon-graphical objects from the CAD file may also be displayed. Forexample, a non-graphical object in the CAD file may represent the word“Shelf”, and the non-graphical object may be associate with thegraphical object associated with the feature 610. By associating thelocation of the non-graphical object with the location of the graphicalobject in the CAD file, and by rendering the graphical object, it ispossible to render the word “Shelf” with the feature 610.

According to some embodiments, non-graphical objects in the CAD file maybe used as meta data (i.e. data that describes other data) in the map.For example, the name “Assembly Area” may be associated with the feature616, even though the text “Assembly Area” may not necessarily bedisplayed on the rendering of the electronica map 600. In this way, thename “Assembly Area” can be used as a human-readable navigationalinstruction, such as by instructing a vehicle to “delivery payload atassembly area”.

Referring to FIG. 7, there is shown a method 700 for electronicallymapping a facility according to some embodiments. According to someembodiments, the method, and individual steps of the method, may berepresented by computer-executable instructions stored on anon-transitory computer-readable medium that, when executed, cause acomputer processor to be configured to execute the method 700 orindividual steps thereof.

The method begins at step 710, when a CAD file is obtained. According tosome embodiments, a CAD file may be uploaded to a computer system, forexample, via a computer network, or using removable storage such as aremovable hard drive or USB thumb drive. According to some embodiments,a CAD file may be uploaded to a server or workstation that is remotelyconnected to sensors. For example, in the case of a self-drivingvehicle, the CAD file may be uploaded to a fleet-management system, or aworkstation used for generating a map file. According to someembodiments, the CAD file may be uploaded to a computer that is local to(i.e. directly connected to) the sensors. For example, in the case of aself-driving vehicle, the CAD file may be uploaded to the vehicle'scontrol system.

At step 712, a keyframe graph is generated based on the CAD file.According to some embodiments, the CAD file may comprise graphicalobjects and non-graphical objects; and the generation of the keyframegraph may be based on the graphical objects. According to someembodiments, generating the keyframe graph may include determiningkeyframe poses based on the detection range of the sensor.

At step 714, an occupancy-map image is generated based on the CAD file.According to some embodiments, the CAD file may comprise graphicalobjects and non-graphical objects; and the generation of theoccupancy-map image may be based on the graphical objects. According tosome embodiments, generating the occupancy-map image may include arasterization of the CAD file.

As used here, “rasterization” refers to the general process ofconverting procedural descriptions into a raster graphic format. In thefield of computer graphics, the term “rasterization” may specificallyrefer to the conversion of a vector graphic to a raster graphic, such asin rasterizing a three-dimensional vector model to a two-dimensionalplane in order to display the resulting two-dimensional raster graphicon a computer display screen. However, as used here, the term“rasterization” is given a more general meaning. For example,“rasterization” can refer to the process of converting graphicalrepresentations in a binary or human-readable format (such as ASCIItext) into a raster image (such as a bitmap). According to someembodiments, the occupancy-map image may be in the form of a bitmapfile.

According to some embodiments, the occupancy-map image may be in theform of a portable greymap (“PGM”) file. According to some embodiments,the bitmap values (i.e. values associated with each pixel) may be avalue corresponding to any one of a state of “occupied”, “unoccupied”,and “unknown”.

Depending on the particular implementation of the system executing themethod 700, the occupancy-map image may be generated by any one of aremote server (e.g. fleet-management system), a remote workstation, anda computer local to a sensor such as that of a control system on aself-driving vehicle. For example, in some cases, the occupancy-mapimage may be subsequently used by a computer local to a sensor such asthat of a control system on a self-driving vehicle, but theoccupancy-map image may nonetheless be (originally) generated by aremote computer based on a CAD file. In other cases, a computer local toa sensor such as that of a control system on a self-driving vehicle maygenerate the occupancy-map image based on a CAD file.

At step 716, a map file is stored on a non-transitory computer-readablemedium. According to some embodiments, the map file is based on the CADfile, the keyframe graph, and the occupancy-map image, as previouslydescribed. The map file may be stored on any or all of a remote computersuch as a server (e.g. fleet-management system) or workstation, and acomputer local to a sensor such as that of a control system of aself-driving vehicle.

At step 718, the map file may be rendered and displayed. According tosome embodiments, the map file may be rendered and displayed by any orall of a remote computer (e.g. a workstation) and a computer that islocal to a sensor. For example, in the case of a self-driving vehicle(i.e. the sensors are on the vehicle), a local display may be mounted onor near the vehicle in order to provide a user interface to a humanoperator for reporting on the mapping and location of vehicle and/orproviding tasks and instructions to the vehicle. Similarly, a remoteworkstation may display the map file in order to provide a userinterface to a human operator in a location that is different from thevehicle's location in order to report on the mapping and location of thevehicle and/or provide tasks and instruction to the vehicle.

As previously described, rendering and displaying the map file maycomprise any or all of rendering and displaying the individual CAD file,which may include graphical and non-graphical objects, and theoccupancy-map image. According to some embodiments, rendering anddisplaying the map file may comprise rendering and displaying theoccupancy-map image, wherein the occupancy-map image is based oninformation from the CAD file and/or information obtained from sensorscans. According to some embodiments, rendering and displaying the mapfile may comprise rendering and displaying graphical objects directlyfrom the CAD file and rendering and displaying the occupancy-map image(e.g. with one rendering overlaying the other). According to someembodiments, rendering and displaying the map file may comprise atextual rendering of non-graphical objects from the CAD file, which mayinclude overlaying the textual renderings with the graphical renderingsbased the graphical objects from the CAD file and/or the occupancy-mapimage.

Generally, the graphical objects of the CAD file and the occupancy-mapimage represent spatial features (e.g. defining a particular feature interms of its spatial relationships and positioning), whereas thenon-graphical objects of the CAD file represent features that are notnecessarily defined in terms of space. As such, according to someembodiments, associations between graphical objects and non-graphicalobjects, as defined in the CAD file, can be used to spatially locate thenon-graphical objects, if desired. For example, if the non-graphicalobjects represent text, and it is desired to render the text whenrendering the map file, then the text can be spatially located inrelation to the spatial features based on the associations defined inthe CAD file.

At step 720, a feature is detected by a sensor. For example, the featuremay be detected by a sensor of a self-driving vehicle. Generally, thedetection of a feature may be associated with a particular pose of thesensor.

At step 722, a determination is made as to whether the detected featureis a new feature, or a known feature. In other words, was the detectedfeature included in the CAD file obtained during step 710 and/or has thefeature been detected during a previous iteration of the method 700?According to some embodiments, the determination can be made bycomparing the sensor scan with which the feature was detected with thecurrent occupancy-map image. If the sensor scan indicates that aparticular pixel is “occupied” but the occupancy-map image indicates“unoccupied” or “unknown”, then the detected feature may be a newfeature.

If, at step 722, it is determined that the feature is a changed feature(and thus not a known feature), then the method proceeds to step 724. Atstep 724, the occupancy-map image is updated based on the difference infeatures. As used here, the term “changed feature” can refer to any orall of an addition, alteration, or removal of a feature. For example, anew feature may be added, and this may be a “changed feature”. A knownfeature may be removed, and this may be a “changed feature”. A knownfeature may be moved or reoriented, and this may be a “changed feature”.

According to some embodiments, the keyframe graph is a part of (i.e.stored in) the map file, and, therefore, updating the keyframe graph issynonymous with updating the map file. According to some embodiments,the keyframe graph may be subsequently updated accordingly. According tosome embodiments, the map file may be updated and the keyframe graph maybe updated. As such, the “changed” feature may become a feature in akeyframe that maybe be subsequently used to locate the sensor relativeto the map according to known SLAM techniques.

According to some embodiments the occupancy-map image is a part of (i.e.stored in) the map file, and, therefore, updating the occupancy-mapimage is synonymous with updating the map file. According to someembodiments, the occupancy-map image may be updated, and the map filemay be subsequently updated accordingly. According to yet furtherembodiments, the map file may be updated and then the occupancy-mapimage updated. As such, the “changed” feature becomes a “known” feature.The method then proceeds to step 726. According to some embodiments,determining that a feature is a changed feature may be via known SLAMtechniques.

If, at step 722, it is determined that the feature is a known feature(and thus not a new feature), then the method proceeds to step 726.

At step 726, the sensor may be located with respect to the detected(known) feature. According to some embodiments, a localization techniquemay be employed according to known SLAM techniques. According to someembodiments, the localization technique may use one or more keyframes asa reference, using SLAM techniques such as visual SLAM or integratedinertial-visual SLAM.

According to some embodiments, locating the sensor (e.g. locating avehicle that includes a sensor) within a facility may comprise comparinga feature detected by the sensor with the keyframe data associated withthe keyframe graph. Since each keyframe is associated with a pose, whena match is found between the keyframe data and the feature detected bythe sensor, the location of the sensor can be determined based on thepose associated with the matched keyframe.

At step 728, the sensor may be moved to a new pose in order to perform anew scan. For example, if the sensor is a sensor on a self-drivingvehicle, the vehicle may move through a facility in the performance of aparticular task or mission (e.g. a task or mission unrelated tomapping). While the vehicle is moving, sensor scans may besimultaneously performed, which may contribute to ongoing mapping (orremapping) of the facility. In this way, a CAD file describing theentire facility may be used to generate an initial occupancy-map image,and occupancy-map image can be updated based on sensor scans in order toaccount for any changes in the facility. As such, the CAD file may beused as an original “map” of the facility without the need for thevehicle to traverse (i.e. “map”) the entire facility. However, despitethe CAD file being static, the “map” may be updated as the vehiclesubsequently traverses the facility.

The present invention has been described here by way of example only.Various modification and variations may be made to these exemplaryembodiments without departing from the spirit and scope of theinvention, which is limited only by the appended claims.

We claim:
 1. A method for electronically mapping a facility, comprising:obtaining a CAD file from at least one non-transitory computer-readablemedium, the CAD file having graphical representations representing knownfeatures of the facility; generating an occupancy-map image based on theCAD file; detecting a sensed feature in the facility with at least onesensor; and updating the occupancy-map image based on the sensedfeature; wherein the known features of the facility do not comprise thesensed feature prior to detecting the sensed feature.
 2. The method ofclaim 1, further comprising: detecting at least one of the knownfeatures of the facility with the at least one sensor; and locating theat least one sensor relative to the occupancy-map image based on thedetecting the at least one of the known features.
 3. The method of claim1, wherein generating the occupancy-map image comprises a rasterizationbased on the CAD file.
 4. The method of claim 3, wherein the CAD filecomprises graphical objects, and the rasterization is based on thegraphical objects.
 5. The method of claim 4, wherein the CAD filefurther comprises non-graphical objects; the method further comprisingstoring a map file on the at least one non-transitory computer-readablemedium, wherein the map file comprises the occupancy-map image, thegraphical objects, and the non-graphical objects.
 6. The method of claim1, further comprising generating a keyframe graph based on the CAD file,and storing a map file on the at least one non-transitorycomputer-readable medium, wherein the map file comprises theoccupancy-map image and the keyframe graph.
 7. The method of claim 1,wherein: the at least one sensor is in communication with a controlsystem of a self-driving vehicle; the method further comprising: storingthe occupancy-map image on the control system of the self-drivingvehicle; and navigating the self-driving vehicle based on theoccupancy-map image.
 8. The method of claim 1, wherein the at least onesensor comprises a LiDAR device.
 9. The method of claim 1, wherein theat least one sensor comprises a vision system.
 10. A method for locatinga sensor in a facility, comprising: obtaining a CAD file from at leastone non-transitory computer-readable medium, the CAD file havinggraphical representations representing known features of the facility;generating a keyframe graph based on the CAD file, the keyframe graphcomprising a plurality of keyframes, each keyframe associated with apose; detecting a sensed feature in the facility with the sensor;comparing the sensed feature with the plurality of keyframes;determining a current pose of the sensor based on matching the sensedfeature with a matched keyframe; wherein the matched keyframe is one ofthe plurality of keyframes, and the current pose is the pose associatedwith the matched keyframe.
 11. The method of claim 10, furthercomprising detecting a new feature in the facility with the sensor, andupdating the keyframe graph based on the new feature, wherein the knownfeatures of the facility do not comprise the new feature prior todetecting the new feature.
 12. The method of claim 10, whereingenerating the keyframe graph comprises determining the plurality ofposes based on the range of the sensor.
 13. A system for electronicallymapping a facility, comprising: at least one sensor for detecting sensedfeatures in the facility; at least one non-transitory computer-readablemedium storing a CAD file having graphical representations representingknown features of the facility; and at least one processor incommunication with the at least one sensor and at the least onenon-transitory computer-readable medium, the processor configured to:generate an occupancy-map image based on the CAD file; store theoccupancy-map image on the at least one medium; and update theoccupancy-map image based on the sensed features; wherein the knownfeatures of the facility do not comprise the sensed features prior tothe sensor detecting the sensed features.
 14. The system of claim 13,wherein: the at least one sensor comprises at least one sensor on aself-driving vehicle; the at least one medium comprises a first mediumthat is part of a control system on the self-driving vehicle; the atleast one processor comprises a first processor that is part of thecontrol system on the self-driving vehicle; and the at least oneprocessor configured to store the occupancy-map image comprises thefirst processor storing the occupancy-map image on the first medium. theat least one processor configured to update the occupancy-map imagecomprises the first processor updating the occupancy-map image on thefirst medium.
 15. The system of claim 14, wherein: the at least onemedium comprises a second medium that is part of a server incommunication with the self-driving vehicle; the at least one processorcomprises a second processor that is part of the server; and the atleast one medium storing the CAD file comprises the second mediumstoring the CAD file; and the at least one processor configured togenerate the occupancy-map image comprises the second processorconfigured to generate the occupancy-map image; wherein the servertransmits the occupancy-map image to the self-driving vehicle.
 16. Thesystem of claim 14, wherein: the at least on medium storing the CAD filecomprises the first medium storing the CAD file; and the at least oneprocessor configured to generate the occupancy-map image comprises thefirst processor configured to generate the occupancy-map image.
 17. Thesystem of claim 13, wherein the at least one sensor comprises a LiDARdevice.
 18. The system of claim 13, wherein the at least one sensorcomprises a vision system.
 19. A method for displaying an electronicmap, comprising: receiving a map file from a non-transitorycomputer-readable medium, the map file comprising graphical objects, anoccupancy-map image associated with the graphical objects, andnon-graphical objects associated with the graphical objects; displayinga rendering of the map file on a computer display, wherein the renderingcomprises a graphical rendering of the occupancy-map image and textualrendering of the non-graphical objects.
 20. The method of claim 19,wherein the graphical rendering of the occupancy-map image comprisesspatial features, and the textual rendering of the non-graphical objectscomprises spatially locating text in relation to the spatial featuresbased on the association between the spatial features, the graphicalobjects, and the non-graphical objects.